Image capture device with adaptive white balance correction using a switchable white reference

ABSTRACT

This disclosure relates to image capture devices with the ability to perform adaptive white balance correction using a switchable white reference (SWR). In some embodiments, the image capture device utilizes “true white” information to record images that better represent users&#39; perceptions. In other embodiments, the same SWR and camera that dynamically sample ambient lighting conditions are used to determine “true white” in near real-time. In other embodiments, the image capture device comprises a display screen that utilizes the “true white” information in near real-time to dynamically adjust the display. In other embodiments, face detection techniques and/or ambient light sensors may be used to determine which device camera is most closely-aligned with the direction that the user of the device is currently looking in, and using it to capture a “true white” image in the direction that most closely corresponds to the ambient lighting conditions that currently dominate the user&#39;s perception.

BACKGROUND

This disclosure relates generally to the field of image capture, andmore particularly, to image capture devices with the ability to performadaptive white balance correction using a switchable white reference.

The advent of portable integrated computing devices has caused awidespread proliferation of cameras and video devices. These integratedcomputing devices commonly take the form of smartphones or tablets andtypically include general purpose computers, cameras, sophisticated userinterfaces including touch sensitive screens, and wirelesscommunications abilities through Wi-Fi, LTE, HSDPA and other cell-basedor wireless technologies. The widespread proliferation of theseintegrated devices provides opportunities to use the devices'capabilities to perform tasks that would otherwise require dedicatedhardware and software. For example, as noted above, integrated devicessuch as smartphones and tablets typically have two or more embeddedcameras. These cameras generally amount to lens/camera hardware modulesthat may be controlled through the general purpose computer usingfirmware and/or software (e.g., “Apps”) and a user interface includingthe touch-screen fixed buttons and touchless control such as voicecontrol.

The integration of cameras into communication devices such assmartphones and tablets has enabled people to share and view images andvideos in ways never before possible. It is now very popular to acquireand immediately share photos with other people, either by sending thephotos via text message, SMS, or email, or by uploading the photos to anInternet-based service, such as a social networking site or a photosharing site.

Most portable integrated computing devices also incorporate at least onedisplay screen to exchange information with users. Images may becaptured by one or more cameras integrated with the device and displayedon the device's display screen, along with other content. During dailyusage, users may experience numerous different ambient lightingconditions. The human eye and brain automatically adapt to the ambientlighting environment and process what is seen to be “color correct.”However, electronic devices are still largely agnostic to ambientlighting condition changes, which can cause problems that users canperceive.

One common problem relates to the fact that white balance is partiallytaken care of by the camera when images are captured, but the correctionprocess is not perfect, and recorded images can often be tinted. In suchcases, the recorded image does not accurately represent what the useractually perceived at the moment the image was captured. Thus, an objectmay be perceived by the user as perfectly “white” at the moment of imagecapture, but recorded by the camera as cyan-tinted.

A second problem relates to the fact that, other than reflectivedisplays that utilize natural ambient lighting as the light source, allelectronic devices utilize some type of internal light source. As aresult, the images displayed on screen are often rendered agnostic ofthe ambient lighting conditions. Thus, the device's screen may have acorrect physical white point, i.e., the emitted spectrum is supposed toproduce correct white, however, if the user has been adapted to theparticular lighting conditions in the ambient environment, the colors onthe device's screen may not appear to be rendered correctly to the user.For example, in an ambient environment that has red-deficient lightingconditions, the device's screen may appear particularly reddish to theuser.

Many studies have been conducted attempting to determine methods toallow automatic white balancing on cameras and post-processing of theimages. A typical white balancing algorithm involves determining whichpart of image is “true white” and adjusting the remainder of the imagebased on the determined “true white” portion of the image. Typicalchoices of “true white” may be a shiny highlight spot on an objectcaused by specular reflection or large areas of objects that arerecorded by the image sensor as having a color that is close to white.

However, there are limitations on how much a camera can correct thewhite balance of an image based on the captured images. The whitebalance correction process relies on the objects in the image providingenough relevant color information for the software to find a “truewhite.” Often, professional photographers shoot at a standard whitereference first before taking photos of the targets in order to get anaccurate white balance. As for the displays on consumer electronicdevices, little has been done to satisfactorily correct the problem ofadaptive white balance correction.

SUMMARY

Described herein are various methods, devices and computer readablemedia utilizing an image capture device with adaptive white balancecorrection capabilities. In particular, some embodiments describedherein comprise an image capture device comprising one or moreswitchable white references (SWR) that enable the device's camera(s) tocapture accurate “true white” images.

In other embodiments described herein, an image capture device isdisclosed that utilizes the “true white” information to record imagesthat better represent users' perceptions. In still other embodimentsdescribed herein, the same SWR and camera that dynamically sampleambient lighting conditions are used to determine “true white” in nearreal-time. In still other embodiments described herein, an image capturedevice comprises a display screen that utilizes the “true white”information in near real-time to dynamically adjust the image-on-screen.

In yet other embodiments, face detection techniques and/or ambient lightsensors may be used to determine which device camera (e.g., in the caseof devices with multiple cameras facing different directions) is mostclosely-aligned with direction that the user of the device is currentlylooking in, and using it to capture a “true white” image in order tocapture a “true white” image in the direction that most closelycorresponds to the ambient lighting conditions that are currentlydominating the user's field of view/perception.

Further embodiments include methods and non-transitory program storagedevices, readable by a programmable control device and comprisinginstructions stored thereon to cause one or more processing units toimplement the functionality described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary image capture device having afront-facing camera and a rear-facing camera, in accordance with oneembodiment.

FIG. 2 illustrates an exemplary switchable white reference (SWR) design,in accordance with one embodiment.

FIG. 3A illustrates an exemplary switchable white reference comprising apolymer-dispersed liquid crystal display (PDLC), in accordance with oneembodiment.

FIG. 3B illustrates an exemplary switchable white reference comprising areverse mode polymer stabilized cholesteric texture (PSCT), inaccordance with one embodiment.

FIG. 3C illustrates an exemplary switchable white reference comprisingan electro-wetting display, in accordance with one embodiment.

FIG. 4A illustrates an exemplary process for utilizing an SWR to performwhite balancing in flowchart form, in accordance with one embodiment.

FIG. 4B illustrates another exemplary process for utilizing an SWR toperform white balancing in flowchart form, in accordance with oneembodiment.

FIG. 4C illustrates yet another exemplary process for utilizing an SWRto perform white balancing in flowchart form, in accordance with oneembodiment.

FIG. 5 illustrates yet another exemplary process for utilizing an SWR toperform white balancing in flowchart form, in accordance with oneembodiment.

FIG. 6 illustrates an exemplary process for performing chromaticadaptation transformation in flowchart form, in accordance with oneembodiment.

FIGS. 7A-7D illustrate exemplary scenarios in which a multi-cameradevice with an SWR and face detection capabilities may be utilized toemploy improved white balancing techniques, in accordance with someembodiments.

FIG. 7E illustrates an exemplary process for utilizing an SWR and facedetection with a multi-camera device to perform white balancing inflowchart form, in accordance with one embodiment.

FIG. 8 illustrates a simplified functional block diagram of anillustrative electronic image capture and display device, according toone embodiment.

DESCRIPTION

Systems, methods and program storage devices are disclosed, whichprovide instructions to cause one or more cameras and/or processingunits to utilize a switchable white reference to perform improved whitebalance correction for images. The techniques disclosed herein areapplicable to any number of electronic devices with cameras anddisplays, such as: digital cameras, digital video cameras, mobilephones, personal data assistants (PDAs), portable music players,monitors, as well as desktop, laptop, and tablet computer displays.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the inventive concept. As part of this description,some of this disclosure's drawings represent structures and devices inblock diagram form in order to avoid obscuring the invention. In theinterest of clarity, not all features of an actual implementation aredescribed in this specification. Moreover, the language used in thisdisclosure has been principally selected for readability andinstructional purposes, and may not have been selected to delineate orcircumscribe the inventive subject matter, resort to the claims beingnecessary to determine such inventive subject matter. Reference in thisdisclosure to “one embodiment” or to “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one implementation of theinvention, and multiple references to “one embodiment” or “anembodiment” should not be understood as necessarily all referring to thesame embodiment.

It will be appreciated that, in the development of any actualimplementation (as in any development project), numerous decisions mustbe made to achieve the developers' specific goals (e.g., compliance withsystem- and business-related constraints), and that these goals may varyfrom one implementation to another. It will also be appreciated thatsuch development efforts might be complex and time-consuming, but wouldnevertheless be a routine undertaking for those of ordinary skill in thedesign of an implementation of image processing systems having thebenefit of this disclosure.

Referring now to FIG. 1, an exemplary image capture device 100 (frontside)/150 (rear side) having a front-facing camera 104 and a rear-facingcamera 152 is illustrated, in accordance with one embodiment. Theexemplary image capture device 100/150 may also comprise other typicaluser interface-related features, such as a device display screen 102,one or more physical buttons 108, and a front-facing ambient lightsensor 106. As mentioned above, the rear-side 150 of exemplary imagecapture device 100/150 may comprise a rear-facing camera 152, an ambientlight sensor 154 and one or more flash elements 156 to aid in theprocess of image capture. In other embodiments, a greater number ofcameras, sensors, flash elements, and/or cameras facing in additionaldirections may be included in the design of the image capture device.

Referring now to FIG. 2, an exemplary switchable white reference (SWR)design 162/166 is illustrated, in accordance with one embodiment. Thedashed-line box 175 illustrates a magnified schematic view of theexemplary SWR design. As shown in FIG. 2, the rear-facing camera 152 ofthe device is mounted between an outer housing 160 and cover glass layer158. The SWR 162/166 is shown as being mounted between the rear-facingcamera 152 and cover glass layer 158.

The SWR 162/166 is a switchable device that may be configured to changebetween a “transparent mode” (150 a) and a “diffusive white mode” (150b). In the transparent mode, the SWR 162 allows incident light rays 164to pass through without scattering. In the diffusive white mode, the SWR166 generates strong scattering of the incident light rays 168, whichthen serve as the reference or “true white” color recorded by the imagesensor of camera 152. The SWR 162/166 may be placed directly in front ofthe camera 152 or to the side, where an optical path may be made forlight rays to be diffused by the SWR before getting into the camera. Insome embodiments the SWR 162/166 may actually be integrated into thecover glass layer 158 in order to provide a wider aperture and collectincident light rays from a wider angle. In still other embodiments, theSWR 162/166 may be physically rotated or swung into and out of positionin front of the camera 152 by electromechanical elements, such asgear(s), motor(s), magnet(s), hinge(s), etc.

There are many possible mechanisms by which the SWR may be implemented,several of which will now be discussed in further detail below withreference to FIGS. 3A-3C.

Referring first to FIG. 3A, an exemplary switchable white referencecomprising a polymer-dispersed liquid crystal display (PDLC) isillustrated, in accordance with one embodiment. The PDLC may comprisesof PET film (300), and a polymer filler material (304) surrounded by aconductive coating (306). When the power from voltage source 308 isturned on (as shown with the switch in position 310), the liquid crystalmolecules (302) become arranged in a regular manner, allowing light topass through uninterrupted, i.e., the “transparent mode.” By contrast,when the power from voltage source 308 is turned off (as shown with theswitch in position 314), the liquid crystal molecules (312) becomearranged in an irregular manner, thus dispersing the incident lightrays, i.e., the “diffusive white mode.”

Referring now to FIG. 3B, an exemplary switchable white referencecomprising a reverse mode polymer stabilized cholesteric texture (PSCT)is illustrated, in accordance with one embodiment. In the PSCT exampleshown in FIG. 3B, when voltage source 326 is powered off, incident lightrays 324 a are transmitted through the liquid crystals 320 a and polymernetwork 322 uninterrupted, i.e., the “transparent mode” referred toabove. By contrast, when voltage source 326 is powered on, incidentlight rays 324 b are scattered during transmission through the liquidcrystals 320 a and polymer network 322, i.e., the “diffusive white mode”referred to above.

Referring now to FIG. 3C, an exemplary switchable white referencecomprising an electro-wetting display is illustrated, in accordance withone embodiment. Electro-wetting, or “electrophoretic” display technologyworks by utilizing electricity to cause liquids that are blended withwhite particles that are designed to scatter incident light to migrate,due to wetting properties changes of the commanding surface 346. Asshown in FIG. 3C, “white mode” (340 a) comprises a transparent liquid352, a white liquid 344 a trapped between two cell walls 350, acommanding surface 346, and a transparent electrode 348. In the “whitemode” configuration, the white liquid 344 a covers the aperture ofcamera 152, causing it to capture a “true white” image. By contrast,“transparent mode” (340 b) comprises a voltage source 354 that ispowered on to cause the white liquid 344 b to migrate along thecommanding surface 346 towards one of the cell walls 350, such that thewhite liquid 344 b no longer covers the aperture of camera 152, causingit to capture a non-scattered, i.e., non-white, image.

Referring now to FIG. 4A, an exemplary process 400 for utilizing an SWRto perform white balancing is illustrated in flowchart form, inaccordance with one embodiment. As will be explained in greater detailbelow, in process 400, the “true white” image is taken before the “real”image that the user wishes to capture. First, the user depresses thecamera's “shutter button” (or equivalent user interface element) toindicate the desire to capture an image using the camera (Step 402).Next, the SWR is configured to be in the “diffusive white mode” (Step404). At that point, a “true white” image may be captured (Step 406),from which near real-time white balance information may be extracted(Step 416). Next, the SWR is configured to be in the “transparent mode”(Step 408), so that a normal image may be captured (Step 410). Thecaptured image may then be processed at Step 412 with the added benefitof the white balance information from Step 416, resulting in a correctedimage 414, which may be stored in memory and/or presented to the user,e.g., via the device's display.

Referring now to FIG. 4B, an exemplary process 418 for utilizing an SWRto perform white balancing is illustrated in flowchart form, inaccordance with one embodiment. In process 418, the “true white” imageis taken after the “real” image that the user wishes to capture. First,the user depresses the camera's “shutter button” (or equivalent userinterface element) to indicate the desire to capture an image using thecamera (Step 420). Next, the SWR is configured to be in the “transparentmode” (Step 422). At that point, the normal image is captured (Step424). Next, the SWR is configured to be in the “diffusive white mode”(Step 426) so that a “true white” image may be captured (Step 428), fromwhich near real-time white balance information may be extracted (Step434). The captured image may then be processed at Step 430 with theadded benefit of the white balance information from Step 434, resultingin a corrected image 432, which may be stored in memory and/or presentedto the user, e.g., via the device's display.

Referring now to FIG. 4C, an exemplary process 436 for utilizing an SWRto perform white balancing is illustrated in flowchart form, inaccordance with one embodiment. In process 436, a “true white” image istaken both before and after the “real” image that the user wishes tocapture. First, the user depresses the camera's “shutter button” (orequivalent user interface element) to indicate the desire to capture animage using the camera (Step 438). Next, the SWR is configured to be inthe “diffusive white mode” (Step 440). At that point, the “before” “truewhite” image may be captured (Step 442), from which near real-time whitebalance information may be extracted (Step 456). Next, the SWR isconfigured to be in the “transparent mode” (Step 444). At that point,the normal image is captured (Step 446). Next, the SWR is againconfigured to be in the “diffusive white mode” (Step 448) so that the“after” “true white” image may be captured (Step 450), from whichadditional near real-time white balance information may be extracted(Step 456). The captured image may then be processed at Step 452 withthe added benefit of both the “before” and the “after” white balanceinformation from Step 456, resulting in a corrected image 454, which maybe stored in memory and/or presented to the user, e.g., via the device'sdisplay. Various techniques may be employed to combine the white balanceinformation taken from both the “before” and “after” “true white”images. For example, the “before” and “after” images may be averaged,interpolated, or combined in some other fashion to produce a weightedaverage, e.g., giving greater weight to the “true white” image with ahigher average brightness value or giving greater weight to the “truewhite” image that was taken closer in time to the “real” image capturedby the user.

Referring now to FIG. 5, an exemplary process 500 for utilizing an SWRto perform white balancing is illustrated in flowchart form, inaccordance with one embodiment. According to process 500, the SWRautomatically switches between a “diffusive white mode” (512 a/512 b/512n . . . ) and a “transparent white” mode (514 a/514 b/514 n . . . ) oncea user enters a “camera” mode or application on the device. A series 516of “true white” reference images (516 a/516 b/516 n . . . ) may thus berecorded before and/or after the “real” image captured by the user. Theprocess for capturing the “real” image is similar to that described inFIGS. 4A-4C: the user depresses the camera's “shutter button” (orequivalent user interface element) to indicate the desire to capture animage using the camera (Step 502); the SWR is configured to be in the“transparent mode” (Step 504); the “real” image is captured (Step 506);and the captured image may then be processed at Step 508 with the addedbenefit of the white balance information from Step 518, resulting in acorrected image 510, which may be stored in memory and/or presented tothe user, e.g., via the device's display.

The device may use all the white balance information gathered from theseries 516 of “true white” reference images (516 a/516 b/516 n . . . )to intelligently figure out the best white balance gains for all of thereference white images, resulting in extracted white balance information518. This technique may be particularly useful if the response time ofthe SWR is not fast enough to avoid any noticeable delay in taking aphoto using the previously described processes.

The process illustrated in FIG. 5 may be extended further and employedeven when users are not intending to take a photo. For example, thedevice camera(s) and SWR can be turned on from time to time, e.g., atpredetermined time intervals, in order to capture a series of true whitereference images. The information collected from the true whitereference images may then be used by the device to determine what kindof ambient light conditions the device is in. This information may thenbe used to adjust the white balance of the content displayed on thedevice's display screen. In addition, this information may provideadditional information to the device's ambient light sensor(s) withregard to the overall brightness of the environment. The information mayalso be used to adjust display brightness and other settingsaccordingly, based on the ambient lighting conditions gleaned from thereference white images.

By employing the above-described techniques, the images recorded by thecamera should better represent user's perception. However, when therecorded content and/or other system-generated content is displayed onthe device's display screen, the user of the device could be in totallydifferent lighting conditions to which he or she is alreadywell-adapted. Ideally, the emitted display white point should be “colorcorrect” (e.g., for a D65 light source) regardless of ambient lightingconditions. However, when the user looks at the same displayed contentunder fluorescent lighting conditions and incandescent lightingconditions, the content's appearance could look totally different due tothe chromatic adaption to the particular ambient lighting conditions. Asa result, displayed content should be modified, e.g., via a chromaticadaptation transformation, to recreate the correct perception, i.e., theuser's perception at the time when the content was generated.

Referring now to FIG. 6, an exemplary process 600 for performing such achromatic adaptation transformation is illustrated in flowchart form, inaccordance with one embodiment. First, a series of “true white”reference images may be captured (Step 614), as described above withreference to Step 516 of FIG. 5. Next, ambient condition parametersincluding, e.g., ambient light correlated color temperature (CCT) andlight intensity, will be directly extracted by analyzing the referencewhite images (Step 616). In the meantime, the original input image 602may go through a de-gamma block (Step 604) to convert from the digitalcount domain to luminance domain (e.g., CIEXYZ). Next, a chromaticadaptation transformation (CAT) may be applied, taking into account theambient condition parameters to predict the corresponding colors underthe current ambient conditions (Step 606). The CAT may then be followedby an inverse color space transformation (Step 608). The image data maythen be converted back to the RGB linear domain, and finally been-gammed (Step 610) to re-generate the corrected image (Step 612) sothat the colors in the image will look “perceptually correct.”

One exemplary CAT is the CIE Chromatic Adaptation Transformation 02(CIECAT02). Given a set of tristimulus values in XYZ, the correspondingRGB value in LMS space is calculated by M_(CAT02) to prepare foradaptation, as shown in Eqns. 1 and 2 below:

$\begin{matrix}{\begin{bmatrix}R \\G \\B\end{bmatrix} = {M_{{CAT}\; 02}\begin{bmatrix}X \\Y \\Z\end{bmatrix}}} & \left( {{Eqn}.\mspace{14mu} 1} \right) \\{M_{{CAT}\; 02} = \begin{bmatrix}0.7328 & 0.4296 & {- 0.1624} \\{- 0.7036} & 1.6975 & 0.0061 \\0.0030 & 0.0136 & 0.9834\end{bmatrix}} & \left( {{Eqn}.\mspace{14mu} 2} \right)\end{matrix}$

The D factor of degree of adaptation, see Eqn. 3 below, is a function ofthe surround (F=1, average, F=0.9, Dim, F=0.8, Dark), and L_(A) is theadapting field luminance in cd/m². D can be set to zero for noadaptation and unity for complete adaptation.

$\begin{matrix}{D = {F\left\lbrack {1 - {\left( \frac{1}{3.6} \right){\mathbb{e}}^{(\frac{{- L_{A}} - 42}{92})}}} \right\rbrack}} & \left( {{Eqn}.\mspace{14mu} 3} \right)\end{matrix}$

Given the D factor, and data transformed using M_(CAT02), the fullchromatic adaptation transform can be written as shown in Eqn. 4 below,where the w subscript denotes the corresponding value for referencewhite point and the c subscript denotes stimuli values. G_(c) (green)and B_(c) (blue) adapted values may be calculated in a similar manner.R _(c)=[(Y _(w) ^(D) /R _(w))+(1−D)]R  (Eqn. 4)

Finally, the corresponding color can be calculated by multiplying theinverse chromatic adaptation transformation matrix to go back to theCIEXYZ domain, as shown in Eqn. 5 below.

$\begin{matrix}{\begin{bmatrix}X \\Y \\Z\end{bmatrix} = {M_{{CAT}\; 02}^{- 1}\begin{bmatrix}R_{C} \\G_{C} \\B_{C}\end{bmatrix}}} & \left( {{Eqn}.\mspace{14mu} 5} \right)\end{matrix}$

Referring now to FIGS. 7A-7D, exemplary scenarios in which amulti-camera device with an SWR and face detection capabilities may beutilized to employ improved white balancing techniques are illustrated,in accordance with some embodiments.

In FIG. 7A, the user is looking at the device display on the front ofthe device, and the rear-facing side of the device is facing away fromthe user. In some such embodiments, the device may determine thedirection of the user's gaze by running a face detection/facerecognition algorithm using the front-facing camera and/or a rear-facingcamera. If the user of the device's face is detected looking at thefront-facing camera of the device, the device may determine that thedevice's rear-facing camera should be used to capture the reference“true white” images for the white balance correction process(represented by dashed lines 701), since the rear-facing camera is mostclosely aligned with the direction that the user is currently lookingin, and thus the ambient lighting conditions that are likely to bedominating the users' current environment. Conversely, if the user ofthe device's face is detected looking at the rear-facing camera of thedevice, the device may determine that the device's front-facing camerashould be used to capture the reference “true white” images for thewhite balance correction process.

In FIG. 7B, the user is looking at the device display on the front ofthe device and using the rear-facing camera of the device to take apicture in the direction that the user is looking. In some suchembodiments, when the device is in such a ‘camera’ mode using thedevice's rear-facing camera, the device may determine that the device'srear-facing camera should be used to capture the reference “true white”images for the white balance correction process (represented by dashedlines 702), since the rear-facing camera is most closely aligned withthe direction in which the user is currently attempting to capture animage, and thus the ambient lighting conditions that are likely todominate the image that the user is about to capture.

In FIG. 7C, the user is looking at the device display on the front ofthe device while the device is laying on a flat surface. In some suchembodiments, the device may determine that the device's rear-facingcamera is blocked and/or otherwise reading a very low ambient lightlevel and thus should not be used to capture the reference “true white”images for the white balance correction process, instead using thedevice's front-facing camera (represented by dashed lines 703). In otherembodiments, if the device is not in a ‘camera’ mode and no user face isdetected, the device may utilize whichever device camera is reading ahigher ambient light level to capture the reference “true white” images.

In FIG. 7D, the user is looking at the device display on the front ofthe device and using the front-facing camera of the device to take aself-portrait or “selfie” picture of himself. In some such embodiments,when the device is in such a ‘camera’ mode using the device'sfront-facing camera, the device may determine that the device'sfront-facing camera should be used to capture the reference “true white”images for the white balance correction process (represented by dashedlines 704), since the front-facing camera is most closely aligned withthe direction in which the user is currently attempting to capture animage, and thus the ambient lighting conditions that are likely todominate the image that the user is about to capture.

Referring now to FIG. 7E, an exemplary process 700 for utilizing an SWRand face detection with a multi-camera device to perform white balancingis illustrated in flowchart form, in accordance with one embodiment.First, a white balance correction request is received at the device(Step 705). This request could come either from the user pressing thecamera shutter button (as described above with reference to FIGS.4A-4C), or merely from a predetermined amount of time passing, e.g., ifthe device is periodically sampling the white level of the device'sambient environment (as described above with reference to FIG. 5).

At that point, the device can detect the device orientation (e.g., inwhich directions are each of the device's cameras facing) using internalsensors, such as gyrometers and/or accelerometers (Step 710). The devicecan also detect what type of operational mode the device is in, e.g., a‘camera mode,’ such as front-facing camera mode or rear-facing cameramode, display mode, eBook reading mode, etc. (Step 710). When thisinformation has been ascertained, the SWR may be configured to be in the“diffusive white mode” (Step 715). Next, if the device is in ‘cameramode’ (Step 720), the process may proceed to: capture the “true white”image with the device camera that is currently being utilized by thedevice's ‘camera mode’ (Step 725); extract the relevant white balanceinformation (Step 765); configure the SWR to be in “transparent mode”(Step 730); capture the “real” image (Step 735); and then perform imageprocessing (Step 740) with the added benefit of the white balanceinformation from Step 765, resulting in a corrected image (Step 745).

If, instead, at Step 720, the device determines that it is not in‘camera’ mode, it may execute a face detection/face recognitionalgorithm using the front-facing camera and/or rear-facing camera. If aface is detected, the device may then use whichever of the device'scameras is more closely aligned with the direction the user is currentlylooking in (assuming that the device camera most closely aligned withthe direction the user is currently looking in is also reading greaterthan a minimum threshold ambient light level) (Step 750) to capture thereference “true white” images for the white balance correction process(Step 755), from which the relevant white balance information may beextracted (Step 765). The process may then proceed to perform imageprocessing (Step 740) on the display of the device with the addedbenefit of the white balance information from Step 765, resulting in acorrected device display (Step 745). The reason for the minimumthreshold ambient light level requirement is for situations such as thatshown in FIG. 7C, wherein the camera most closely aligned with thedirection the user's face is currently looking in may be against a flatsurface, and thus provide little or no meaningful ambient light levelinformation.

If, instead, at Step 750, a face is not detected or the thresholdambient light level is not met by the relevant device camera, the devicemay then use whichever of the device's cameras is reading a greaterambient light level to capture the reference “true white” images for thewhite balance correction process (Step 760), from which the relevantwhite balance information may be extracted (Step 765). The process maythen proceed to perform image processing (Step 740) on the display ofthe device with the added benefit of the white balance information fromStep 765, resulting in a corrected device display (Step 745).

FIG. 8 is a simplified functional block diagram of an illustrativeelectronic device for image capture and display, according to oneembodiment. Electronic device 800 may include processor 805, display810, user interface 815, graphics hardware 820, device sensors 825(e.g., proximity sensor/ambient light sensor, accelerometer and/orgyroscope), microphone 830, audio codec(s) 835, speaker(s) 840,communications circuitry 845, digital image capture unit 850, videocodec(s) 855, memory 860, storage 865, and communications bus 870.Electronic device 800 may be, for example, a personal digital assistant(PDA), personal music player, a mobile telephone, or a notebook, laptopor tablet computer system.

Processor 805 may execute instructions necessary to carry out or controlthe operation of many functions performed by device 800. Processor 805may, for instance, drive display 810 and receive user input from userinterface 815. User interface 815 can take a variety of forms, such as abutton, keypad, dial, a click wheel, keyboard, display screen and/or atouch screen. Processor 805 may be a system-on-chip such as those foundin mobile devices and include a dedicated graphics processing unit(GPU). Processor 805 may be based on reduced instruction-set computer(RISC) or complex instruction-set computer (CISC) architectures or anyother suitable architecture and may include one or more processingcores. Graphics hardware 820 may be special purpose computationalhardware for processing graphics and/or assisting processor 805 processgraphics information. In one embodiment, graphics hardware 820 mayinclude a programmable graphics processing unit (GPU).

Sensor and camera circuitry 850 may capture still and video images thatmay be processed to generate images in accordance with this disclosure.Output from camera circuitry 850 may be processed, at least in part, byvideo codec(s) 855 and/or processor 805 and/or graphics hardware 820,and/or a dedicated image processing unit incorporated within circuitry850. Images so captured may be stored in memory 860 and/or storage 865.Memory 860 may include one or more different types of media used byprocessor 805, graphics hardware 820, and image capture circuitry 850 toperform device functions. For example, memory 860 may include memorycache, read-only memory (ROM), and/or random access memory (RAM).Storage 865 may store media (e.g., audio, image and video files),computer program instructions or software, preference information,device profile information, and any other suitable data. Storage 865 mayinclude one more non-transitory storage mediums including, for example,magnetic disks (fixed, floppy, and removable) and tape, optical mediasuch as CD-ROMs and digital video disks (DVDs), and semiconductor memorydevices such as Electrically Programmable Read-Only Memory (EPROM), andElectrically Erasable Programmable Read-Only Memory (EEPROM). Memory 860and storage 865 may be used to retain computer program instructions orcode organized into one or more modules and written in any desiredcomputer programming language. When executed by, for example, processor805 such computer program code may implement one or more of the methodsdescribed herein.

It is to be understood that the above description is intended to beillustrative, and not restrictive. The material has been presented toenable any person skilled in the art to make and use the invention asclaimed and is provided in the context of particular embodiments,variations of which will be readily apparent to those skilled in the art(e.g., some of the disclosed embodiments may be used in combination witheach other). In addition, it will be understood that some of theoperations identified herein may be performed in different orders. Thescope of the invention therefore should be determined with reference tothe appended claims, along with the full scope of equivalents to whichsuch claims are entitled. In the appended claims, the terms “including”and “in which” are used as the plain-English equivalents of therespective terms “comprising” and “wherein.”

The invention claimed is:
 1. An apparatus, comprising: two or morecameras, each camera comprising an image sensor and a switchable whitereference element configured to switch between a first transparent modeand a second diffusive mode; a display; a memory having, stored therein,computer program code; and one or more processing units operativelycoupled to the memory and configured to execute instructions in thecomputer program code that cause the one or more processing units to:determine a first operational mode of the apparatus; and if the firstoperational mode is not determined to be a camera mode— attempt todetect a face of a user of the apparatus using one or more of the two ormore cameras; and if the face of a user of the apparatus is detected andif the camera most closely aligned with a direction that the detecteduser's face is currently looking in is reading greater than a minimumthreshold ambient light level— configure the switchable white referenceelement of the apparatus camera most closely aligned with a directionthat the detected user's face is looking in into the second diffusivemode; capture a first reference white image using the apparatus cameramost closely aligned with a direction that the detected user's face islooking in; determine a first white point for the first reference whiteimage; and adjust a color composition of the display based, at least inpart, on the determined first white point for the first reference whiteimage.
 2. The apparatus of claim 1, wherein the first transparent modeof a respective camera of the two or more cameras comprises a modewherein light passes through the respective camera's switchable whitereference element to the respective camera's respective image sensorwithout scattering.
 3. The apparatus of claim 1, wherein the seconddiffusive mode of a respective camera of the two or more camerascomprises a mode wherein light passes through the respective camera'sswitchable white reference element to the respective camera's respectiveimage sensor with scattering.
 4. The apparatus of claim 1, wherein theone or more processing units are further configured to executeinstructions in the computer program code that cause the one or moreprocessing units to determine the first operational mode of theapparatus at predetermined time intervals.
 5. The apparatus of claim 1,wherein the switchable white reference element of at least one of thetwo or more cameras comprises at least one of the following: apolymer-dispersed liquid crystal display (PDLC), a reverse mode polymerstabilized cholesteric texture (PSCT), and an electro-wetting display.6. The apparatus of claim 1, wherein the instructions in the computerprogram code further cause the one or more processing units to: if thefirst operational mode is determined to be a camera mode— configure theswitchable white reference element of the apparatus camera beingutilized in the camera mode into the second diffusive mode; capture asecond reference white image using the apparatus camera being utilizedin the camera mode; determine a second white point for the secondreference white image; configure the switchable white reference elementof the apparatus camera being utilized in the camera mode into the firsttransparent mode; obtain a first captured image from the image sensor ofthe apparatus camera being utilized in the camera mode; and adjust acolor composition of the first captured image based, at least in part,on the second determined white point for the second reference whiteimage.
 7. The apparatus of claim 6, wherein the instructions in thecomputer program code further cause the one or more processing units to:if the first operational mode is not determined to be a camera mode, andthe face of a user of the apparatus is detected, and if the camera mostclosely aligned with a direction that the detected user's face iscurrently looking in is not reading greater than a minimum thresholdambient light level, or if the first operational mode is not determinedto be a camera mode, and no face of a user of the apparatus is detected—configure the switchable white reference element of the apparatus camerareading the greatest ambient light level into the second diffusive mode;capture a third reference white image using the apparatus camera readingthe greatest ambient light level; determine a third white point for thethird reference white image; and adjust a color composition of thedisplay based, at least in part, on the third determined white point forthe third reference white image.
 8. A non-transitory program storagedevice, readable by a programmable control device and comprisinginstructions stored thereon to cause one or more processing units to:determine a first operational mode of an apparatus, the apparatuscomprising: two or more cameras, each camera comprising an image sensorand a switchable white reference element configured to switch between afirst transparent mode and a second diffusive mode; and a display; andif the first operational mode is not determined to be a camera mode—attempt to detect a face of a user of the apparatus using one or more ofthe two or more cameras; and if the face of a user of the apparatus isdetected and if the camera most closely aligned with a direction thatthe detected user's face is currently looking in is reading greater thana minimum threshold ambient light level— configure the switchable whitereference element of the apparatus camera most closely aligned with adirection that the detected user's face is looking in into the seconddiffusive mode; cause the apparatus to capture a first reference whiteimage using the apparatus camera most closely aligned with a directionthat the detected user's face is looking in; determine a first whitepoint for the first reference white image; and adjust a colorcomposition of the display based, at least in part, on the determinedfirst white point for the first reference white image.
 9. Thenon-transitory program storage device of claim 8, wherein the firsttransparent mode of a respective camera of the two or more cameras ofthe apparatus comprises a mode wherein light passes through therespective camera's switchable white reference element to the respectivecamera's respective image sensor without scattering.
 10. Thenon-transitory program storage device of claim 8, wherein the seconddiffusive mode of a respective camera of the two or more cameras of theapparatus comprises a mode wherein light passes through the respectivecamera's switchable white reference element to the respective camera'srespective image sensor with scattering.
 11. The non-transitory programstorage device of claim 8, wherein the one or more processing units arefurther configured to execute instructions that cause the one or moreprocessing units to determine the first operational mode of theapparatus at predetermined time intervals.
 12. The non-transitoryprogram storage device of claim 8, wherein the switchable whitereference element of at least one of the two or more cameras of theapparatus comprises at least one of the following: a polymer-dispersedliquid crystal display (PDLC), a reverse mode polymer stabilizedcholesteric texture (PSCT), and an electro-wetting display.
 13. Thenon-transitory program storage device of claim 8, further comprisinginstructions stored thereon to cause the one or more processing unitsto: if the first operational mode is determined to be a camera mode—configure the switchable white reference element of the apparatus camerabeing utilized in the camera mode into the second diffusive mode; causethe apparatus to capture a second reference white image using theapparatus camera being utilized in the camera mode; determine a secondwhite point for the second reference white image; configure theswitchable white reference element of the apparatus camera beingutilized in the camera mode into the first transparent mode; cause theapparatus to obtain a first captured image from the image sensor of theapparatus camera being utilized in the camera mode; and adjust a colorcomposition of the first captured image based, at least in part, on thesecond determined white point for the second reference white image. 14.The non-transitory program storage device of claim 13, furthercomprising instructions stored thereon to cause the one or moreprocessing units to: if the first operational mode is not determined tobe a camera mode, and the face of a user of the apparatus is detected,and if the camera most closely aligned with a direction that thedetected user's face is currently looking in is not reading greater thana minimum threshold ambient light level, or if the first operationalmode is not determined to be a camera mode, and no face of a user of theapparatus is detected— configure the switchable white reference elementof the apparatus camera reading the greatest ambient light level intothe second diffusive mode; cause the apparatus to capture a thirdreference white image using the apparatus camera reading the greatestambient light level; determine a third white point for the thirdreference white image; and adjust a color composition of the displaybased, at least in part, on the third determined white point for thethird reference white image.
 15. A computer-implemented method,comprising: determining a first operational mode of an apparatus, theapparatus comprising: two or more cameras, each camera comprising animage sensor and a switchable white reference element configured toswitch between a first transparent mode and a second diffusive mode; anda display; and if the first operational mode is not determined to be acamera mode— attempting to detect a face of a user of the apparatususing one or more of the two or more cameras; and if the face of a userof the apparatus is detected and if the camera most closely aligned witha direction that the detected user's face is currently looking in isreading greater than a minimum threshold ambient light level—configuring the switchable white reference element of the apparatuscamera most closely aligned with a direction that the detected user'sface is looking in into the second diffusive mode; causing the apparatusto capture a first reference white image using the apparatus camera mostclosely aligned with a direction that the detected user's face islooking in; determining a first white point for the first referencewhite image; and adjusting a color composition of the display based, atleast in part, on the determined first white point for the firstreference white image.
 16. The computer-implemented method of claim 15,wherein the second diffusive mode of a respective camera of the two ormore cameras of the apparatus comprises a mode wherein light passesthrough the respective camera's switchable white reference element tothe respective camera's respective image sensor with scattering.
 17. Thecomputer-implemented method of claim 15, wherein the one or moreprocessing units are further configured to execute instructions thatcause the one or more processing units to determine the firstoperational mode of the apparatus at predetermined time intervals. 18.The computer-implemented method of claim 15, wherein the switchablewhite reference element of at least one of the two or more cameras ofthe apparatus comprises at least one of the following: apolymer-dispersed liquid crystal display (PDLC), a reverse mode polymerstabilized cholesteric texture (PSCT), and an electro-wetting display.19. The computer-implemented method of claim 15, further comprising: ifthe first operational mode is determined to be a camera mode—configuring the switchable white reference element of the apparatuscamera being utilized in the camera mode into the second diffusive mode;causing the apparatus to capture a second reference white image usingthe apparatus camera being utilized in the camera mode; determining asecond white point for the second reference white image; configuring theswitchable white reference element of the apparatus camera beingutilized in the camera mode into the first transparent mode; causing theapparatus to obtain a first captured image from the image sensor of theapparatus camera being utilized in the camera mode; and adjusting acolor composition of the first captured image based, at least in part,on the second determined white point for the second reference whiteimage.
 20. The computer-implemented method of claim 19, furthercomprising: if the first operational mode is not determined to be acamera mode, and the face of a user of the apparatus is detected, and ifthe camera most closely aligned with a direction that the detecteduser's face is currently looking in is not reading greater than aminimum threshold ambient light level, or if the first operational modeis not determined to be a camera mode, and no face of a user of theapparatus is detected— configuring the switchable white referenceelement of the apparatus camera reading the greatest ambient light levelinto the second diffusive mode; causing the apparatus to capture a thirdreference white image using the apparatus camera reading the greatestambient light level; determining a third white point for the thirdreference white image; and adjusting a color composition of the displaybased, at least in part, on the third determined white point for thethird reference white image.